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ABSTRACT 



This paper addresses how to measure the performance of 
programs by measuring the outcomes of program participation for learners. 
Based on studies conducted in Tennessee in which adult learners reported a 
broader and more complex set of outcomes than the three "core indicators” 
mandated by Title II of the 1998 Workforce Investment Act (WIA) , the paper 
suggests that learners have a different perspective on performance than the 
authors of WIA and their perspectives should be considered at the policy 
level and by local programs. Section 1 examines performance accountability in 
adult basic education and how it is being applied in the WIA, the National 
Reporting System, and 17 state plans. Section 2 reports on findings of the 
Tennessee Longitudinal Study of Adult Literacy and a subsequent qualitative 
study. The following findings of the Tennessee study of adult literacy 
learners are reported: (1) there is an increase in rate of employment; (2) 

increase in overall satisfaction with their financial situation; (3) overall 
increase in self esteem; (4) increase in involvement in community 
organizations; (5) positive change in three of eight literacy practices; and 
(6) increase in the number of people who thought a book was a good gift for a 
child. Findings from the Learner Identified Outcomes Study show that 
participants identified changes in their lives that can be grouped as 
literacy uses and changes in sense of self. Section 3 discusses implications 
of findings and alternative ways to assess the performance of adult basic 
education. (Contains 17 references.) (YLB) 
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Outcomes of Participation in Adult Basic Education: 
The Importance of Learners’ Perspectives 



Introduction 

This paper addresses an issue of concern to adult educators across the United States: how to 
measure the performance of programs by measuring the outcomes of program participation for 
learners. All federally funded programs will soon begin measuring three “core indicators” 
mandated by Title II of the 1998 Workforce Investment Act (WIA). Measurement of these core 
indicators will vary from state to state, but all states will use the National Reporting System 
(NRS) developed by the Division of Adult Education and Literacy (DAEL) of the Department of 
Education’s Office of Vocational and Adult Education. 

Based on studies conducted in Tennessee in which adult learners reported a broader and more 
complex set of outcomes than the WIA core measures, this paper suggests that learners have a 
different perspective on performance than the authors of WIA and that their perspectives should 
be taken into account at the policy level as well as by local programs. While these adults did 
report outcomes, including increased employment, that correspond with the WIA indicators, they 
were more likely to name outcomes related to their sense of self and to changes in how they used 
literacy in their everyday lives. The challenge for the adult basic education system is to develop 
measures of program performance that credit these more complex and nuanced changes. 

The paper begins by examining performance accountability in adult basic education and how it is 
being applied in the Workforce Investment Act and the National Reporting System. We then 
report on the findings of a longitudinal study of Tennessee adult literacy learners and a 
subsequent qualitative study. Finally, implications of these findings and some alternative ways to 
assess the performance of adult basic education are discussed. 

Performance Accountability in Adult Basic Education 

The current focus in adult basic education on measuring outcomes is part of a larger concern 
with performance that permeates government. There are many ways to judge the successful 
performance of a program. Performance accountability systems focus on judging policies and 
programs by measuring their outcomes or results against agreed-upon standards (Brizius and 
Campbell, 1991), rather than on, for example, the number of people served or services offered. 

In her recent review of performance accountability in adult education, Juliet Merrifield discussed 
the “dramatic changes of emphasis which have brought the accountability debate to center stage” 
(Merrifield, 1998, p. 4). These include: 

> Societal changes such as increased globalization, concerns about workforce 
competitiveness and immigration, and welfare reform. 

1 
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> Governmental changes in response to demands for more efficiency and "better return 
on investment”, including the Governmental Performance and Results Act that 
requires federal agencies to establish and report on long-term and annual goals. 

> Changes in the K-12 educational system in response to economic and societal change, 
including Goals 2000 and a push for family literacy. 

^ A definition of literacy in terms ot function and competencies. 

Merrifield concludes that “The social impacts of literacy appear to be the guiding purpose for 
public investment in literacy education" (p. 12). 

Achieving social policy goals has been among the purposes for adult education in the United 
States for many years. Quigley (1997) reviews the social purposes for which literacy education 
has been used: to integrate new immigrants, to teach moral lessons, as a tool in the War on 
Poverty, and today, to combat crime and strengthen the economy. However, while there have 
been many social policy goals for adult literacy education in the past, programs were not 
evaluated on whether or not they met these goals. 

The 1991 National Literacy Act defined literacy as "an individual’s ability to read, write, and 
speak English, and to compute and solve problems at levels ot proficiency necessary to function 
on the job and in society” and also included the more individual purposes of “to achieve one’s 
goals, and develop one’s knowledge and potential" (National Literacy Act of 1991, sec. 3). 

While the 1991 law addressed social policy goals, defining literacy in terms of adults’ need to 
“function,” the legislation also included the idea that people might want education for their own 
reasons and value knowledge for its own sake. Performance of adult education programs under 
the National Literacy Act was assessed more in terms of program quality than by outcomes in 
learners’ lives. 



Assessment of performance changed in the 1998 Workforce Investment Act. Under WIA the 
social policy goals are specified; adult education services are authorized in order to assist adults 
to become employed, be involved in their children’s education, and gain a secondary credential 
(GED or high school diploma). Programs are held accountable for meeting these goals. Instead 
of assessing programs by the quality of their curricula and instruction, support for professional 
development, or breadth of programming, states are required to set up performance 
accountability systems and assess adult basic education programs on their success at achieving 
particular policy outcomes. 

Workforce Investment Act 

The purpose of the Adult Education and Family Literacy Act, Title II of the Workforce 
Investment Act, is to provide adult education services in order to: 

(1) assist adults to become literate and obtain the knowledge and skills necessary for 
employment and self-sufficiency; 

(2) assist adults who are parents to obtain the educational skills necessary to become 
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full partners in the educational development of their children; and 

(3) assist adults in the completion of a secondary school education. (P.L. 105-220) 

Language about meeting personal goals or developing knowledge, which was included in the 
1991 legislation, is absent in WIA. 

WIA mandates a performance accountability system in order "to assess the effectiveness of 
eligible agencies in achieving continuous improvement of adult education and literacy activities 
funded under this subtitle, in order to optimize the return on investment of Federal funds in adult 
education and literacy activities” (P.L. 105-220). States are required to set levels of performance 
for three core indicators: 

(i) Demonstrated improvements in literacy skill levels in reading, writing, and speaking 
the English language, numeracy, problem solving, English language acquisition, and 
other literacy skills. 

(ii) Placement in, retention in, or completion of postsecondary education, unsubsidized 
employment or career advancement. 

(iii) Receipt of a secondary school diploma or its recognized equivalent. (P.L. 105-220) 

The WIA performance accountability system is based on measures of progress in relation to 
these three core indicators. In their Adult Education and Family Literacy plans submitted to the 
federal Division of Adult Education and Literacy, states have established performance targets for 
each indicator. Performance on these core indicators will be reported by the states annually, 
using the National Reporting System described below. States will be responsible for collecting 
data on the core indicators and reporting the aggregated data to the Department of Education. 

The states may choose to identify additional indicators, but these will not be used to assess 
performance at the Federal level. 

The WIA accountability system operates on two levels. On the federal level, the Division of 
Adult Education and Literacy will use data on the core indicators in their reports to Congress on 
the effectiveness of adult education, to publish state-by-state comparisons, and as part of 
determining states’ eligibility for certain incentive grants established under WIA. States will use 
the core indicators along with other state-identified factors in assessing program performance 
and in determining allocation of funds. 

National Reporting System 

The National Reporting System for Adult Education has been developed by the American 
Institutes for Research under contract to the Division of Adult Education and Literacy. This 
system establishes measures for the core indicators required by the Workforce Investment Act. 
The development of the NRS began as the National Outcomes Reporting System Project. That 
project (begun before WIA was enacted) was developed in response to the concerns of state 
directors of adult education about performance accountability. The original system was to focus 
on seven categories of outcomes endorsed by state directors: economic impact, credentials. 
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learning gains, family impact, further education and training, community impact, and customer 
satisfaction (Condelli & Kutner, 1997). 

The National Reporting System now being implemented focuses on the three core indicators. The 
measures to be reported include: educational gains, entering or retaining employment, and 
placement in postsecondary education or passing the GED test (DAEL / NRS, 1999). In addition 
to the outcome measures, the NRS will collect data on descriptive measures (student 
demographics, reasons for enrolling, and student status) and participation (contact hours and 
enrollment in special programs). 

As can be seen in the NRS graphic reproduced below, the attainment of the goals of the Adult 
Education and Family Literacy Act is to be measured in large part by the achievement of basic 
academic skills that are measured in most states by standardized tests. The limited connection 
between goals and indicators has happened in part because Congress did not establish core 
indicators for family literacy. And while the purposes of WlA refer to the “knowledge and skills 
necessary” for employment and self-sufficiency, these were not defined. 



Goals and Core Indicators of WIA 

Adult Education and Family Literacy Act and NRS Core Outcome Measures 



Goals of Adult Education 
Described in the Adult 
Education and Family 
Literacy Act of WIA 


Core Indicators Required by 
the Adult Education and 
Family Literacy Act of WIA 


National Reporting System 
Core Outcome Measures 


Assist adults to become 
literate and obtain the 
knowledge and skills 
necessary for employment and 
self-sufficiency. 

Assist parents to obtain the 
skills necessary to be full 
partners in their children’s 
educational development. 


Improvements in literacy 
skill levels in reading, 
writing and speaking the 
English language, numeracy, 
problem-solving, English 
language acquisition, other 
literacy skills. 


• Educational gains 

(achieve skills to advance 
educational functioning 
level) 


Placement in, retention in, or 
completion of, postsecondary 
education, training, 
unsubsidized employment or 
career advancement. 


• Entered employment 

• Retained employment 

• Placement in 
postsecondary education 
or training 


Assist adults in the completion 
of secondary school 
education. 


Receipt of a'secondary 
school diploma or its 
recognized equivalent. 


• Receipt of a secondary 
school diploma or pass 
GED test 



(NRS, 1999, p. 6) 



4 




9 



NCSALL 



January 2000 



Educational gains are to be measured by movement from one to another of six “educational 
functioning levels” which include: Beginning ABE Literacy, Beginning Basic Education, Low 
Intermediate Basic Education, High Intermediate Basic Education, Low Adult Secondary 
Education, High Adult Secondary Education. The English as a Second Language (ESL) levels 
are: Beginning ESL Literacy, Beginning ESL, Low Intermediate ESL, High Intermediate ESL, 
Low Advanced ESL, and High Advanced ESL (DAEL / NRS, 1999, p.6). For each level the 
NRS has identified descriptors in three areas: Basic Reading and Writing, Numeracy Skills, and 
Functional and Workplace Skills for ABE; and Speaking and Listening, Basic Reading and 
Writing, and Functional Workplace Skills for ESL. For example, the ability to complete medical 
forms is a functional skill at the Low Intermediate Basic Education Level, and reading text about 
and explaining the use of a complex piece of machinery is a workplace skill at the High Adult 
Secondary Level. The ability to participate in a conversation in some social situations is a 
descriptor for speaking and listening in the High Intermediate ESL level. 

Placement of adult learners in these levels is conducted at enrollment and again after pre- 
determined time periods while the adult is an active student. All programs in a state must use the 
same assessment procedure. While the NRS states "Their inclusion in no way is meant to imply 
that the tests are equivalent or that they should be used as the basis for assessment ” [italics in 
original] (DAEL / NRS, 1999, p. 19), test benchmarks are provided for each educational 
functioning level. The NRS assumes that increases in scores on standardized tests like the Test of 
Adult Basic Education (TABE) or Comprehensive Adult Student Assessment System (CASAS) 
will indicate increases in skills and, therefore, in functioning levels. 

Employment and postsecondary education and training measures and receipt of a secondary 
credential are follow-up measures to be collected after students leave a program. These 
measures are collected only for those adults whose goals include employment, further education, 
or a secondary credential. States may use either survey or data matching (using Social Security 
numbers) methodologies for the follow-up measures for employment and secondary credentials. 

State Plans 



The Workforce Investment Act mandates three core indicators of performance, and the Division 
of Adult Education and Literacy requires use of the National Reporting System for reporting on 
the indicators beginning July 1, 2000. However, the states will have some flexibility in how these 
requirements are implemented. States have a choice of measure of educational gain; they may 
use a standardized test like TABE or CASAS or “a performance assessment with standardized 
scoring protocols” (DAEL / NRS, 1999, p. 13). This choice will have implications beyond the 
process of reporting. Performance-based assessments can be tied more directly to learners’ goals 
for adult basic education and may give a better indication of how programs are meeting the three 
goals of Title II of WIA. However, these assessments are not in common use, nor identified in 
the NRS. 
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States also have the option of using the “secondary measures” established by the NRS. These 
measures are similar to the ones that the state directors endorsed in earlier development of the 
reporting system: reduction of public assistance, passing the citizenship examination, voting or 
registering to vote for the first time, more involvement in community groups or activities, 
increased involvement in children’s literacy activities and/or education. The NRS suggests using 
these measures only for learners who have these goals. There is also a measure of goal 
accomplishment for so-called “project learners” who have a limited (30 hours of instruction) 
goal of a specific workplace-related literacy skill. While these measures are part of the NRS and 
can be reported to the federal government, they will not be used to assess performance under 
WIA at the federal level. States may use these secondary measures as well as. others they may 
identify as part of their state performance accountability system. 

In order to gain a clearer picture of how states are assessing their own performance, the authors 
have informally surveyed the DAEL's regional coordinators and examined seventeen state plans. 
What we have found is summarized in the following table. These results are not final since state 
plans are being revised, but most states are using standardized tests as the educational gains 
assessment measure. About half the states have included measures of performance in addition to 
the WIA core indicators. 



Table 1 . State Plans Summarized 



Region 


Assessment 


Additional indicators 


Midwest: ND, SD, NE, KS, 
MN, lA, MO, WML, Ml, 
IN, OH 


Standardized tests, 
including TABE, 
CASAS, ABLE. 


One state uses additional indicators in the areas 
of computer literacy and family literacy. 


West: AZ, NM, WA, OR, 
CO, CA, UT,NV, ID, MT, 
WY, AK, HI 


All states are using 
primarily CASAS. 


Five states use additional indicators. 


South: AL, LA, NC, OK, 
TN, TX, WV, KY, AR, SC, 
VA, GA, FL, MS 


TABE is used most 
often, also ABLE and 
CASAS. One state 
mentions using 
portfolio assessment. 


Ten states use additional indicators: student 
retention; meeting individual learner goals; 
enrollment; staff development; offering support 
services and computer programs; children’s 
school readiness in family literacy programs. 


East: NY, VT,NH, ME, 
MA, CT, RI,NJ, DE, MD, 
DC, PA, PR 


Different types of 
standardized tests 
are used as well as 
alternative assessment 
in some states. 


Eleven states use additional indicators: 
recruitment; professional development; support 
services; meeting student goals; attendance; 
vote registration; removal from public 
assistance; family literacy; customer 
satisfaction; improved health practices; level 
completion; further education; higher wages 
and benefits; technology. 
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Adult basic education programs are being asked to be accountable for their performance in 
meeting the policy goals of Congress. But the purpose of adult basic education as defined by 
Congress in WIA and the performance accountability system being developed in response to 
WIA are much narrower than the goals of many adult learners. This system will essentially 
measure gains on standardized tests, employment status, secondary credential, and entry to post- 
secondary training. The outcomes on which performance will be assessed do not include all of 
the particular, contextually-determined outcomes that learners in our research have described. 



Outcomes reported by learners 

Two studies recently conducted in Tennessee suggest that additional outcome measures (and 
methodologies) may be needed to capture the whole picture. The Longitudinal Study of Adult 
Literacy Participants in Tennessee collected data from 1992 to 1995 and included baseline 
interviews with 450 adult students who enrolled in beginning ABE classes. Key findings 
include an increase in rate of employment and various changes in literacy practices. The 
subsequent Learner Identified Outcomes study was based on qualitative life history interviews 
conducted in 1997 and 1998 with ten adults who had originally been part of the longitudinal 
study. These adults reported new and expanded literacy uses in many areas of their lives and 
positive changes in their sense of self 

The Tennessee Longitudinal Study 

The primary objective of the Tennessee Longitudinal Study of Adult Literacy Participants study 
was to expand understanding of how participation in literacy programs changes adults’ quality 
of life. The study focused on four main areas: 

• socio-economic well-being (jobs, income, survival) 

• social well-being (family and community life) 

• personal well-being (self esteem, life satisfaction) 

• physical well-being (health and access to health care) 

Adult learners from nine ABE Level-1 programs from across the 95 counties of the state of 
Tennessee were interviewed for this study. Participants in three cohorts (1992, 1993, 1994) 
were given a baseline interview as they enrolled in ABE programs. All had scored below the 
sixth-grade level on the ABLE (Adult Basic Learning Exam) reading test. Those who had 
indicated they were willing, participated in follow-up interviews at approximately one-year 
intervals. The study was designed to continue for five years and include qualitative (individual, 
program, and community case studies) as well. as quantitative data. However, funding was 
interrupted, and the study was not completed. The interviews that were conducted are 
summarized in Table 2. 
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Two earlier reports (Merrifield, Smith. Rea. and Shriver, 1993. and Merrifield, Smith, Rea, and 
Crosse, 1994) examined the composition of Cohorts One and Two and changes found in the first 
follow-up interviews with Cohort One. .At that point analysis was suspended. In 1999 a report 
(Bingman, Ebert, & Smith, in press) examining the changes found in the first follow-up 
interviews from the combined three Cohorts (N=I 99) was completed. 

Table 2. Tennessee l.onuitudirial Study Interv'iews 





Cohort 1 


Cohort 2 


Cohort 3 


Total 


Baseline 


133 


149 


168 


450 


Follow-up 1 


70 


64 


65 


199 


Follow-up 2 


45 


39 




84 


Follow-up 3 


35 






35 



The questionnaire used in this study included 1 16 questions. To examine socio-economic well- 
being, respondents were asked about their employment and sources of income, as well as about 
other activities used to make ends meet. Social well-being was examined through questions 
relating to family and community involvement. The personal well-being of respondents was 
examined with questions pertaining to self-esteem and lifestyle. The Rosenberg Self-Esteem 
Scale, which examines self-esteem, was selected because it has been widely used with different 
populations and because it is one of the shorter self-esteem instruments. To examine physical 
well-being, students were asked about their health and access to health care. In addition to these 
major areas, respondents were asked general demographic questions. The follow-up 
questionnaires included many of the same questions as the baseline questionnaire, as well as 
some new ones that asked people to reflect on changes in their lives. For the 1999 report, the 56 
questions common to baseline and follow-up questionnaires were analyzed. 

From the combined cohorts, 199 adults participated in a follow-up interview. The demographic 
information for these subjects is summarized in Table 3. 

Table 3. Longitudinal Study Follow-up One Participants 



Gender 


Age 


Race 


Married 


Grade completed 




male 


female 


16-24 


25-44 


45+ 


white 


black 


other 


yes 


no 


Less 
than 6"’ 


6'"-9"’ 


10 -12th 


urban 


rural 


43% 


57% 


18% 


57% 


25% 


60% 


38% 


2% 


46% 


54% 


12% 


53% 


35% 


47% 


53% 
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Readers should be aware of the limitations of this research. The sample size was small, and 
learner attrition over time resulted in further decrease in sample sizes. All the data in the study 
were self-reported. There was no comparison group in this study. It was beyond the means of the 
research team to obtain a representative sample of adults who would be eligible for the study but 
did not enroll in ABE (a current longitudinal study being conducted by NCSALL will have a 
comparison group.) Within these limitations, the study provides learners’ responses to a wide 
variety of potential outcomes and contributes to our understanding of the impacts of ABE 
participation in learners’ lives. 

In the analysis of the follow-up interviews of the combined cohorts, a number of important 
changes were found. Approximately one year after enrollment in adult literacy classes these 
adults reported changes in various aspects of their lives. These include: 

• An increase in rate of employment from 32% to 48%. 

• An increased overall satisfaction with their financial situation in Cohorts 2 and 3 (p<.05). 

• An overall increase in self-esteem (as measured by the Rosenberg Self-Esteem Scale) from 
3.52 to 3.66 on a 5 point scale (p<.01). 

• Increased involvement in community (religious, PTA, social/sports) organizations (p<.05). 
There was also a marginally significant increase in voter registration (p <.06). 

• Positive changes in three of eight literacy practices examined (paying bills, working with 
numbers on the job, needing to memorize because of limited reading ability) (p<.05). 

• Increase in the number of people who thought a book was a good gift for a child (p<.05). 

Some of these changes, reported after one year and often limited hours of participation (available 
for 189 of the participants — 60 had more than 80 instructional hours and 129 had less), do not 
seem to represent major transformations in learners’ lives. But small changes in trajectory can 
lead to a major change of direction. These small changes, in combination and over time, may 
well lead to more substantial changes as adult learners expand the scope and level of the ways 
they use literacy. 

The Learner Identified Outcomes Study 



The Learner Identified Outcomes study used a life history methodology to build an 
understanding of the outcomes of participation in adult literacy education on the lives of adult 
learners. The ten participants were selected from those Longitudinal Study participants who had 
at least 80 instructional hours and had a follow-up interview in 1995 (the last year of the 
longitudinal study). This small sample was constructed to be as representative demographically 
of the Tennessee ABE population as possible. 
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Table 4. Participants in Learner Identified Outcomes Study 



Name 


Gender 


Race 


Loca- 

tion 


DOB 


Employment facts 


High school or equivalent 


Grade 

completed 


Bert 


m 


b 


Urban 


1970 


Some past employment, mostly 
temporary. 


To go to military 


9 


Elizabeth 


f 


b 


Urban 


1933 


Long-term employment in a 
restaurant (same employer). Now 
retired. 


Wanted to get education, 
important to me 


7 


Harry 


m 


w 


Small 

town 


1924 


Long-term employment with in 
manufacturing with two employers. 
Now retired. 


Always wanted a diploma 


8 


June 


f 


w 


Urban 


1966 


Little past employment. Now is 
working as a kitchen aide 


Get GED to get a job and 
job training 


9 


Kris 


f 


w 


Small 

town 


1968 


Some past employment as an 
assistant manager and in chicken 
processing plant. 


Better job, get off AFDC 
& Food Stamps 


9 


Laura 


f 


b 


Urban 


1952 


Past employment in service jobs and 
manufacturing. 


To learn spelling, reading, 
math, to better self, 
support family 


12 


Marvin 


m 


b 


Rural 


1945 


Long-term w/ same employer 
(truck-driver & mechanic). Now 
retired due to health. 


To learn to read and write, 
improve basic skills and 
self as citizen 


5 


Ruth 


f 


w 


Small 

town 


1955 


Long-term employment in food 
processing 


GED to get better job 


8 


Suzanne 


f 


w 


Small 

town 


1966 


Past employment in manufacturing. 
Resumed work last year. 


Get GED to go to factory 


10 


Will 


m 


w 


Small 

town 


1945 


Long-term employment in 
manufacturing. 


Better job & benefits; to 
prove I can do it 


8 



The primary source of data for this study was extended recorded conversations with the ten 
participants about their lives before and after enrollment in adult literacy programs. The 
interviews, usually conducted in participants’ homes, covered their adult education experiences, 
their family and work lives, their childhood and earlier schooling, and the changes in their lives 
that they attributed to adult education participation. 

The data from the interviews were analyzed using an inductive iterative process. We noted both 
particular stories and common themes and categories. The broad categories that cut across all 
interviews were, to some extent, determined by the questions asked, e.g. work, adult education, 
early schooling, and family. We also identified themes that emerged from the interviews that cut 
across categories, e.g. value of education, impact of poverty and race on education, literacy 
practices, and sense of self 
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This study explored the lives of these ten adults as well as their definitions of outcomes of 
participation in adult basic education. Their lives established the contexts for the outcomes. All 
of the participants had been employed, many at the same job for many years. Only one person 
was currently dependent on public assistance. Nine of the participants had children, and all 
reported being involved in their children’s education. Of their 1 8 adult children, all but one 
completed high school and eight had attended college. Eight of their children are still in school. 
Many of these people had already met the goals of WlA for self-sufficiency and partnership in 
their children’s education. However, their participation in adult basic education was important 
to them (four had re-enrolled when interviewed) and led to positive changes in their lives. 

Study participants identified a variety of changes in their lives that they attributed to their 
participation in adult literacy education. Some of the adults in this study reported outcomes that 
correspond to the goals of WIA. Two of them passed the GED test, gaining a secondary 
credential. Of the seven who had employment as a goal, four were employed when interviewed. 
However, they reported many other changes that were particular to the individuals and the 
contexts of their lives. These were the changes that seemed to make the most difference in their 
lives, and these changes are grouped here as “literacy uses” and “changes in sense of self” 

Literacy Uses 

Nine of the participants reported acquiring new literacy skills from their participation in adult 
literacy programs. For eight, these new skills in reading, writing, and computation led to 
changes in the ways they use literacy in their lives. Some of the changes were in the practical 
everyday activities of peoples’ lives: 

Fix my own money order out. I fill all my own money order and everything. [Elizabeth] 

/ think I write more [letters to family] now. And I think that 1 write better. Before I just 
wrote everything straight. I did not even know what a paragraph was. And sometimes I 
used words that shouldn't even be there in that paragraph. Now that I know about it, I 
may use dijferent words. [Ruth] 

And also the adult classes taught me how to read a map. I learned how to find myself 
around here in town with the little yellow map book when I had to go out and look for 
apartments. [Laura] 

Some were able to carry out work functions more easily: 

But while I went down there, [to class] it really made a difference, you know, like on 
where I work now, I fill out, you ’ve got to fill out your tickets on what you run, you got 
to keep the account on it, you ’ve got to go through all this and fill it out, date it and what 
it is and all that. And it ’s helped me on all of that. [Will] 
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While most of those with children had already been involved in their children's education, new 
literacy skills expanded this involvement for some: 

My little gnindbaby now, I've learned so many things that I try to teach him a lot of 
things that I could not teach him when he first came to us. [Marvin] 

In addition to the new uses of literacy in carrying out die activities of their lives, the participants 
also described increased access to and understanding of expository text. Five people talked about 
the more extensive reading they now do and how that reading has expanded their understanding 
of the world or themselves. 

You see, the news talks about what's going on over. seas and stuff. You [I] read the hook, 
and I can understand what's going on over there now. Before, I didn't know. I thought, 
well, that's just news, something to report, that's it. Then after I read the hook and 
learned how this become, I understood more. I was, "OK. now this is how this ended up 
at. "I’d he walking in there to get something, and something about overseas happen [on 
television]. I ’d stop and come hack in here. "Wait a minute now, I 've got to catch this. 
Grow interested now. I understand more now. [June] 

7 learned to read my Bible even better and to form my more closer relationship with the 
Lord. [Laura] 

Some of the changes in literacy uses that were reported by the adults in this study included new 
uses; for example, Elizabeth's ability to purchase her own money orders. Others, like Will, were 
better able to carry out activities such as completing job reports that they had previously found 
troublesome. For several, reading became an activity that is a part of their life instead of a tool 
used with difficulty. Will, for example, went from occasionally glancing at newspapers to 
reading them as "an every night thing." Changes in literacy uses are related to changes in 
people’s lives, changes that expand what they are able to do, what they are concerned with, and 
how they feel about themselves. 

Sense of self 

The adults in this study are in many ways ordinary adults: they had had jobs, had raised families, 
are involved in community activities. They are people who are resilient and who have a strong 
sense of their own abilities and their self-efficacy. Even so, these adults also described positive 
changes in their sense of self that they attributed to their participation in adult literacy programs. 
Three participants talked about losing their sense of shame at being in a literacy class: 
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And I was shamed, that ’s another thing. / eoiildn t .see myself going to class, grown man. 
fifty years old almost, .sitting up in clu.ss. And Just was embarrassing to me. But after I 
started and I .seen more than Just me sitting there, .some people were sixty and seventy 
years old. I said, "Why should / he ashamed'.^ There 's some people older than I am. " 

And, hey, I got more into it. [Marvin] 

Four of the participants, the two who passed the GED test and the two who reported the most 
limited reading skills at enrollment, expressed a strong sense of accomplishment. 

It made a whole lot of differenee how I feel about myself because I feel better about 
myself since I learned how to read better. I feel like I 'm .somebody. You feel better about 
yourself when you learn how to do a lot of things for yourself you know. [Elizabeth] 

Four times cmd I finally, finally done it [pas.sed the GED]. And it was all kinds of 
certificates. I got them all on my wall, you know, and / keep looking at them and think. 
"Well, I did that." [Suzanne] 

A new and stronger voice or new opportunities to express themselves were reported by three 
participants as Ruth described: 

And also about speaking up. I can do this better now. Like recently, at my work. I was 
scheduled to have a vacation for Christmas. And then my supervisor comes to me and 
says that this other woman will get the Christmas week off not me. And / was already 
scheduled, and she even had less seniority. So I .spoke up. I said, "No. it isn't right. I 
want my vacation. " And I got it. And before I was so shy. [Ruth] 

The people in this study described new uses of literacy. These literacy practices were often 
important because of the social situations in which they occurred. Literacy uses and sense of self 
often seemed to intersect. So, for example, Marvin had always met with his friends and talked 
about events; but now he can use the knowledge that he has gained from reading the newspapers 
in these conversations. It is not that Marvin “had low self-esteem” before; but now he feels 
differently because of his improved literacy abilities. He has read articles in the paper that his 
friends have read and "so we can discuss this matter. " June felt confident enough to take a job in 
a nursing home kitchen where she is not only able to read the labels on trays, but has started an 
informal English to Speakers of Other Languages (ESOL) program teaching new words to a 
fellow employee. 

He brings the book in with the English words in it. It helps me and it helps him. I figured. 
"Why not try it?" [June] 

The outcomes of participation in literacy classes described by the adults in this study went far 
beyond new skills or educational gains. They are now able take part in new activities that have 
bettered their lives, and in many instances, the lives of those with whom they interact. 
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Why this matters -- Broadening what we measure 

The Workforce Investment Act is law. and a National Reporting System is being established. 

Why should policy makers and program staffin adult basic education concern themselves with 
what some Tennessee adult learners' had to say about outcomes? Students say participation in 
adult education builds their sense of self As people gain new literacy skills they engage in new 
literacy practices. But what does this mean about measuring program performance? If we want 
our programs to meet the needs of adult learners, it means a great deal. 

The learners in these studies reported changes in their lives that were varied, contextual, and 
inter-related. Measuring changes in educational levels with standardized tests will not give 
programs and policy makers information about these outcomes. The tests are neither broad 
enough nor sensitive enough to capture the changes that matter to learners or to measure the 
performance of programs in supporting these changes. Some changes; such as the increases in 
voter registration and community invoK ement found in the longitudinal study, might be 
identified in those states that use the NRS secondary measures; but many of the changes that 
were reported, for example new literacy practices or changes in sense of self will not be 
identified or counted by the WIA indicators. 

Even when these learners met the measures mandated in the NRS (retained employment, 
obtained a GED) these measures do not reveal the meaning of these changes in context. These 
core indicators do not tell us enough about the reality of people's lives. For example, one 
participant who already had a high school diploma reported she could not spell well enough to do 
jobs for which she was physically able. In her life, a secondary credential did not translate into 
employment. And of the four people in the qualitative study who were employed, three were in 
jobs requiring no more education than the jobs they had had before participating in literacy 
programs. They were as, but no more, self-sufficient as before they enrolled. Achievement of 
mandated measures does not necessarily mean positive changes in learners' lives. 

The core indicators required by the Workforce Investment Act define a limited approach to adult 
literacy education. Adult educators tend to focus efforts on what is measured and reported, 
particularly if funding is tied to it. Are we measuring and focusing on what is important? It is 
not clear that a consensus on desired outcomes has been reached or that everyone, particularly 



‘ The findings of changed literacy practices and more positive sense of self are consistent with those ofearlier studies. From the 
twentv-three studies reviewed by Beder (1999). the following potential outcomes ot participation in ABE programs emerged as 
most conclusive: gains in employment and in earnings; positive intluence on continuing education; self-reported gains in basic 
skills; positive impact on learners' self-image and on involverrient in their children's education. Fingeret and Danin (1991) 
identified improvement in literacy skills and changes in literacy practices, as well as acquiring more positive attitudes and beliefs 
as impacts of participation in the Literacy Volunteers in New York city. Fingeret et oL ( 1994) also found increased self-concept 
and self-confidence, growth of cultural awareness, women's empowerment, and more involvement with children's work to be 
among the outcomes of the two literacy programs in British Columbia. Among the literacy practice outcomes, they describe more 
use of writing, improved job performance and being qualified for better Jobs, and new activities that participants were not able to 
do before. 
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learners, have had a voice in determining what is measured. Adult basic education runs the risk 
of not meeting the needs of those adults whose goals for participation in adult education do not 
include (or go beyond) employment or a secondary credential. Adults whose goals are not being 
addressed by adult education programs will leave. 

Performance accountability — judging programs by results — assumes a consensus on standards 
and goals as well as effective ways to measure outcomes and results. Performance accountability 
is a driving force in adult basic education, but there is no consensus about goals. Effective tools 
for measuring the complex variety of outcomes that learners report, or even the narrower goals of 
WIA, do not exist. 

Adult education programs should be accountable for their performance. But their understanding 
of performance needs to be broad enough to take into account the varied needs and wants of 
adult learners and the varied contexts of their communities. Adult educators need to value the 
achievement of both policy goals and learners' goals. To understand the true impact of their 
programs, adult educators need to pay more attention to the varied outcomes in learners’ lives 
and communities. True assessment of program performance cannot be tied to a few limited 
measures. 

If policy makers and practitioners in adult basic education want to address performance 
measurement in ways that take into account the varied goals and life contexts of adult learners, 
they need a new approach to accountability. There are three promising efforts underway that do 
not depend on a few core indicators and scores on standardized tests: expanded use of Program 
Quality Indicators, local documentation of learner goals and outcomes, and the Equipped for the 
Future initiative. 

Program Quality Indicators 

In the 1991 National Literacy Act, adult basic education programs were required to develop 
indicators of program quality. These varied from state to state and even from program to 
program. Providers defined a quality program and determined what indicators would show that 
they were providing quality program services. The indicators focused on inputs such as teacher 
training and the number of classes offered, on processes like record keeping and professional 
development, and on outputs of the program such as retention rates and gains on standardized 
tests. 

The PQI process did not go far enough and was not sufficiently standardized within and across 
states to allow for the kind of comparisons among programs that WIA attempts to accomplish. 
Program Quality needs to be taken another step. Colleges and universities are accredited based 
on what they offer and the quality of their institutions. Comings and Stein (1991) have 
suggested a similar accreditation process for adult basic education. Reaching consensus on what 
constitutes a quality program and what level of quality is appropriate for adult basic education 
would be a complex, long-term process, but one with which states already have some experience. 
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Documenting Outcomes at the Program Level 

Developing ways to document outcomes on the local program level has been the focus of a 
NCSALL action research project in Tennessee, Virginia, and Kentucky. Three programs are 
working with learners to develop processes to identify and document outcomes in learners’ lives 
Two sites have developed documentation processes baised on learner-identified goals and one is 
developing a process to document learner performance on the Equipped for the Future standard, 
Take Responsibility for Learning. 

For example, in a rural Kentucky program, students determined that they shared the goal of 
supporting their children’s education and identified a variety of indicators of such support, 
including reading to their children, getting them to school every day, and talking with their 
children’s teachers. Parents record these activities on a calendar checklist as a way to document 
the outcomes of support for their children's education. The program hopes to have access to 
children’s report cards (with parents’ cooperation and permission) to assess impact on the 
children’s schooling. The process of working with students to identify their goals and document 
outcomes has led the program staff to integrate these goals into program curriculum. 

The action research teams hope to address both the requirements of their states’ performance 
accountability systems and to measure their programs’ success at meeting student-identified 
goals. The teams are meeting with state staff to explore ways that local documentation could 
become part of performance data reported to the state adult education offices. 

Equipped for the Future 

Equipped for the Future is probably the most ambitious effort to broaden the ways adult basic 
education is assessed. This national standards-based system reform initiative, sponsored by the 
National Institute for Literacy, will provide a common framework for defining, tracking, and 
reporting results to policymakers, as well as to students and their local programs. Developed 
through a multi-year field-based research process, the EFF framework consists of: 

> Four purposes for learning, defined originally by adult learners and validated by a wide 
range of adults 

> Three “maps” that define successful performance of the roles of worker, citizen, and 
family member 

> Thirteen activities that are common across these three roles 

> Sixteen skill standards, derived from the role maps, which provide specific and 
measurable statements of what adults need to know and be able to do, clustered in four 
categories: communications skills, interpersonal skills, decision-making skills, and 
lifelong learning skills 

According to Stein (1999), the EFF framework provides the definition of “results that matter”: 
“If we concentrate on building competence across the standards identified in the framework, we 
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will be building the skills and knowledge adults need to carry out the primary responsibilities of 
their adult roles.” Valid and reliable criterion-referenced assessments are essential for a learning 
system focused on helping adults achieve their goals and be successful in their primary roles 
needs. EFF has entered the next phase of development work, that of shaping an assessment 
framework and assessment tools and instruments that focus on the whole circle of EFF skills. 
This work includes the specific tasks of ; 

> Defining the EFF continuum of performance; 

> Developing a continuum of performance for each EFF standard, with levels that 
benchmark key performances; 

> Developing tools to assess performance of each standard for the range of assessment 
purposes, and 

> Developing a broad “qualifications framework” that focuses on integrated performance 
across standards, with levels that represent real world benchmarks. (Stein, in press) 

EFF holds the promise of being the most systematic approach to measuring the development and 
application of skills that enable adults to carry out their life roles: in other words, to achieve 
desired outcomes that benefit themselves and society as a whole. By defining a set of results or 
outcomes (not an exhaustive set, but a commonly agreed-upon set that enable achievement of 
policy goals), and expressing these results in terms that are measurable, programs can teach to 
these outcomes and measure the progress of students toward achieving them (as well as to other 
student — or program — defined goals). How this works in practice will be seen during the next 
two years as EFF continues its field-based research and development work. 



Conclusion 

The Workforce Investment Act, however limited, provides the basis for the beginning of a 
national performance accountability system for adult education. For the present, this will be the 
system that defines performance for federally-funded adult education programs. The system 
established under WIA should be made as inclusive as possible. As states determine how they 
will measure educational gains for the NRS, they should consider using measures other than 
standardized tests. The ways learners define their goals and describe the changes they have seen 
in their lives should inform state policy makers’ decisions regarding the indicators by which 
program performance will be measured. As they revise their adult education plans, states should 
include a variety of indicators that take into account the many ways in which programs impact 
learners. As the inevitable “disconnects” and contradictions built into WIA begin to become 
apparent, adult educators should be certain that their Congressional representatives are made 
aware of what it not working and why. 

There are many aspects of adult education in the United States that could be improved. But adult 
educators need ways of identifying what is working, as well as what is not. There are promising 
initiatives underway that may enable programs to more fully align the adult education system 
with the lives and goals of learners as well as with the social policy needs of the larger public. 
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An expanded model of performance assessment would begin with indicators from WIA and NRS 
and add indicators of program quality and measures of a wider range of outcomes that draw from 
EFF and investigations such as the ones reported here. Adult learners, adult educators, and adult 
education policy makers must continue to engage in building accountability systems that 
recognize the many, varied contributions education can make in the lives of adult learners. 
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The Mission of NCSALL 

The National Center for the Study of Adult Learning and Literacy (NCSALL) will pursue 
basic and applied research in the field of adult basic education, build partnerships betw'een 
researchers and practitioners, disseminate research and best practices to practitioners, scholars and 
policvmakers. and work with the field to develop a comprehensive research agenda. 

NCSALL is a collaborative effort between the Harvard Graduate School ot Education and 
World Education. The Center for Literacy Studies at The University of Tennessee. Rutgers 
University, and Portland State University are NCSALL's partners. NCSALL is funded by the U.S. 
Department of Education through its Office of Educational Research and Improvement (OERI) and 
OERTs National Institute for Postsecondary Education, Libraries, and Lifelong Learning. 

NCSALL's Research Projects 

The goal of NCSALL’s research is to provide information that is used to improve practice 
in programs that offer adult basic education, English to Speakers of Other Languages, and adult 
secondary education services. In pursuit of this goal, NCSALL has undertaken research projects in 
four areas: (1) learner motivation, (2) classroom practice emd the teaching/learning interaction, (3) 
staff development, and (4) assessment. 

Dissemination Initiative 

NCSALL’s dissemination initiative focuses on ensuring that the results ot research reach 
practitioners, administrators, policymakers, and scholars of adult education. NCSALL publishes a 
quarterly magazine entitled Focus on Basics', an annual scholarly review ot major issues, current 
research and best practices entitled Review of Adult Learning and Literacy: and periodic research 
reports and articles entitled NCSALL Reports and NCSALL Occasional Papers. In addition. 
NCSALL sponsors the Practitioner Dissemination and Research Network, designed to link 
practitioners and researchers and to help practitioners apply findings from research in their 
classrooms and programs. NCSALL also has a web site: 

http://gseweb.harvard.edu/~ncsall 

For more information about NCSALL, please contact: 

John Comings, Director 
NCSALL 

Harvard Graduate School of Education 
101 Nichols House, Appian Way 
Cambridge, MA 02138 
(617) 495-4843 

ncsall(^gse. harvard.edu 
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